Repository navigation
fix(tools): audit ordinary homebrew import names - #6239
Conversation
|
Windows check of head Windows 11 Pro 26220, Ryzen 7 5700X, RTX 4070 (driver 596.21), WinLibs MinGW-w64 GCC 15.2.0 r7, Ninja, Release, Python 3.10/3.14.
Three ps5-payload-dev ELFs that ship inside catalog titles, relinked with
The absent lists match the loader. Each converted executable, run with No regression on NID registries: for PPSA99420, PPSA99002 and PPSA99082 (with One small note on the description: Verdict: confirmed. Ordinary homebrew registries now produce a complete audit instead of exit 2, and NID registries are unchanged. AI-assisted (Claude Code). |
|
Corrected the description: this change relaxes registry-name validation; qualification stripping and literal export lookup were already present and remain unchanged. The code and tested head |
|
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
MrMha
left a comment
MrMha
left a comment
There was a problem hiding this comment.
Reviewed exact head f9f6872b8d293ee0353e2f2dd95c49840da209ad on macOS 27.0.1 arm64 with Python 3.12.14.
tools/tests/test_import_audit.py: 81 tests, 80 pass and one explicit Windows read-only replacement skip. This includes literal export resolution, repeated references, mixed encoded/ordinary symbols, weak unresolved imports, and rejection of empty/whitespace/control-character names.- Negative control: retained the PR tests and restored only
tools/import_audit.pyfromHEAD^. The new literal-import regression fails with exit 2 andinvalid NID 'strcmp'. - Additional end-to-end CLI check used a deliberately synthetic ELF export table containing
strcmpand_Z3foovand a matching registry. Parent exits 2 without a report; this head exits 0 and writes a report with two implemented imports. Adding an unresolved weakmissing_functionmakes the fixed CLI exit 1 with two implemented/one absent, confirming permissive input parsing does not turn missing imports into success. Both CLI cases fail atstrcmpwith the parent.
I found no blocking issue in the symbol validation or literal matching. The extra CLI input is a controlled fixture, not a real title dump. No emulator, title, GPU execution, or combined tree with other open PRs was tested.
AI-assisted review and local testing: OpenAI Codex.
|
@MrMha I verified the overlap with #6431 using Git tree-only merge previews against main The resolution needs both this PR's ordinary-symbol validation and #6431's strict boolean This is source inspection and a merge preview, not a combined test run. No code changed in this pass; the existing Linux/Windows/macOS relinker CI results remain tied to this PR's unchanged head. AI-assisted maintenance: OpenAI Codex. |
karimibrahim07
left a comment
karimibrahim07
left a comment
There was a problem hiding this comment.
I ran this head (f9f6872b8) on a real ordinary-name registry: the websrv Quake homebrew's quake.elf (#6698), relinked with --windows --registry, audited with --libs build/core/libs/libs from main 5d9dcc42d. The validation change works: the registry is read instead of failing at the first name. But the result has a problem:
296 references, 293 unique imports
implemented 0 imports
absent 293 imports
Every import is reported absent, including abort, calloc, strlen, pthread_create and scePadOpen. AnyPS5's libraries export these only under their NIDs (strlen is j4ViWNHEgww in libc.prx, pthread_create is OxhIB8LB-PQ in libkernel.prx). A literal export lookup finds none of them. Of the 293:
- 230 are exported by
libs/under the NID of their name. - the other 63 are
gl*/OSMesa*from the bundledlibOSMesa.so.8, which I didn't pass with--modules.
So for this registry, "no built library exports these, so the loader fails" is wrong for every entry.
Suggestion: when the literal lookup misses an ordinary name, also look up compute_nid(name) (tools/nid_names.py already has it). That's what libkernel's dlsym does in FindSymbol (name first, then Nid::ComputeNid(name, "")), and what #6697 does for the Windows loader. A test with a registry entry strlen and a library export j4ViWNHEgww would cover it. Without it, the audit turns from "can't read the registry" into "everything is missing", which is harder to see as a tool problem.
AI-assisted: yes. I ran the audit above and counted the NID matches against the export tables of the built libraries.
|
@karimibrahim07 Fixed in ef3c83d. The audit now tries the literal export first, then its computed NID, and uses the resolved NID for stub classification while retaining the original import name. The three new regressions fail on the previous head and pass fixed, including I reproduced the diagnostic problem with Mari0 1.6.2's hash-verified registries: the same 561 references / 376 unique imports and built libraries change from 376 absent to 355 implemented, 9 stubs and 12 absent. Both runs still exit 1 for remaining gaps. Inputs and libraries are unchanged. The PR description includes exact provenance, commands and limitations. I did not rerun Quake or execute a homebrew title; #6697's Windows loader work remains separate. The existing #6431 |
|
Updated to Validation on Windows 11 x64 / Python 3.13.7:
The description now records the exact integration, tests, limitations and peer-audit evidence. AI-assisted: Codex. |
|
@ArjenS-LA Confirmed on the same Quake registry (#6698), head
That is exactly the 230 NID matches I counted, and the 63 still absent are all the One follow-up, not blocking: with AI-assisted: yes. I ran the relink, the audits and the tests above. |
What
Registries emitted from ordinary ELF homebrew use literal import names, but the audit accepted only 11-character encoded NIDs. Accept nonempty symbol names without whitespace or control characters. Qualification stripping is unchanged. Literal exports take precedence, with a computed-NID fallback when no literal provider exists; reports retain the original import name. The audit continues to report unresolved imports rather than suppressing them.
Tested
Windows 11 x64 (build 26200), WinLibs MinGW-w64 GCC 15.2.0 r7, CMake 4.3.1, Ninja 1.13.2, Python 3.13.7. Baseline:
d92a8dad390af1807cc91622b924252141816a5a. Branch:fix/audit-named-homebrew-imports. Commit:f9f6872b8d293ee0353e2f2dd95c49840da209ad. Committed-object conventions check passed.The same named-import regression fails on unchanged main (
invalid NID 'strcmp') and passes fixed. The audit suite has 77 passes, three platform/privilege skips, and one excluded fixture that creates a temporary Git commit. Coverage includes literal exports, mangled names, encoded NIDs, duplicate references, unresolved weak imports, and rejection of empty/control-character names.Main integration (October 17, 2026)
Current head:
0eb94a28cd98f588ae7ba1665640bc7e38a287f5onfix/audit-named-homebrew-imports. Integrated main66702f1cffd3646f2d674ed093fde0a85a70761fwith a normal two-parent commit so the existing branch advances without rewriting history. The only manual conflict wasread_registry: ordinary import-name validation and literal-first computed-NID fallback are preserved alongside the strict booleanweakvalidation from the now-merged #6431. The prior merge-order request for #6431 is therefore superseded.Windows 11 x64, Python 3.13.7: the upstream boolean-validation regression fails in seven subcases on unchanged previous head
ef3c83d6becbcd6c282d2d5e33d38385898212f5and passes in the integrated tree. The complete audit suite runs 102 tests: 99 pass and 3 explicitly skip for POSIX permissions or Windows symlink privilege; the separate source-stamp fixture that creates a Git commit is excluded. All 34 NID helper tests pass. The exact committed tree isb6be1e83fb8b40aaacedf2261628f93c010aeb34; committed-object conventions and the diff check against current main pass. No C++ rebuild, full-library tests, title execution or hardware validation was performed for this Python tooling integration. Whitespace warnings in two debt files are inherited unchanged from main and absent from the PR diff against main.The same hash-verified Mari0 1.6.2/websrv v0.34 registries, original inputs and 126 built libraries described below were audited freshly on both versions. Both retain 561 references / 376 unique imports: 355 implemented, 9 stubs and 12 absent; both exit 1 for remaining gaps. All original input, registry and library hashes remain unchanged. The prior conversion flags and limitations below still apply; no new conversion or loader execution is claimed. Commands, hashes and reports:
C:/Users/arjen/dev/AnyPS5/.codex/pr-maintainer/logs/pass-20261017T1734Z/homebrew-evidence.json.Fresh complete inventory: 806 open PRs with complete paginated file lists, using main
66702f1cffd3646f2d674ed093fde0a85a70761f. All six authored changes and all 15 authored pairs merge cleanly after this integration. This PR has 26 clean shared-file comparisons; seven remain inconclusive because #4891, #5021, #5390, #5925, #6111, #6171 and #6314 already conflict with main. Across all authored PRs: 80 clean pair previews, one existing #6115/#6927 conflict, and 26 inconclusive comparisons. No combined runtime or exhaustive cross-file semantic-equivalence claim. Exact peer SHAs and tree-only results are recorded incandidate-merge-audit.jsonin the same evidence directory.Review follow-up (October 16, 2026)
Commit
ef3c83d6becbcd6c282d2d5e33d38385898212f5addresses the ordinary-name lookup review. When no literal provider exists, the audit also searchescompute_nid(import_name). Literal exports still win; reports keep the original name, reference count, weak flag, provider mismatch and throwing-stub classification.Windows 11 x64, Python 3.13.7: the three new regressions fail on unchanged prior head
f9f6872b8d293ee0353e2f2dd95c49840da209adand pass with this commit. The audit suite has 80 passes, 3 platform/privilege skips, and 1 excluded fixture that creates a Git commit. All 29 NID helper tests pass.python tools/check_conventions.py --base origin/mainandgit diff --checkpass. This is a Python tooling change; no C++ rebuild or title execution was performed for this follow-up.The same hash-verified Mari0 1.6.2/websrv v0.34 registries (561 references, 376 unique imports) were audited before and after against the same 126 built Windows libraries from source
b7cedd900d67a94453b8a45eec9b5623532a0441: baseline 0 implemented / 0 stubs / 376 absent; fixed 355 implemented / 9 stubs / 12 absent. Both exit 1 because unresolved imports/dependencies remain.strcmpis now correctly supplied bylibc.prx. Original package files and all libraries remain hash-identical.This follow-up reuses the recorded registries from the original bundle converted with the #6231 relinker,
--windows --registry --trap-sdk-syscall-stubs --module-dir ., with no added standard directory. Registry-producing binary SHA-256:854ebbdbdac8c4fe18d17d4bf4253ac9e3efa9063c6b1c7e81e82e4b0cafba23. The prior conversion's registry/companion hashes were verified before these fresh audit runs. Neither run supplies--modules. The built-library snapshot differs from the official v0.1.2 libraries used in the initial evidence below. No fresh conversion or homebrew execution is claimed; export availability does not prove loader compatibility. #6697 separately changes the Windows runtime loader.Reproducer: run
python tools/import_audit.py output.libOSMesa.so.8.guest.prx.registry.json output.registry.json --libs <built-libraries> --source <matching-core/libs/prx> --json <report>on each audit version. Exact commands, all input/library hashes, source hashes, output JSON and limitations:C:/Users/arjen/dev/AnyPS5/.codex/pr-maintainer/logs/pass-20261016T1134Z/homebrew-evidence.json.Fresh overlap check: 763 open PRs, complete file lists, main
bba8b293bab770d0134c6f9a1759353b3f60ad51; all 36 authored pairs merge cleanly. For this PR: 20 clean shared-file comparisons, 4 inconclusive peers already conflicting with main, and the existing #6431 conflict intools/import_audit.py/read_registryat peer head1490e27fd7240963a1d2d9054c5b28bd2e58bc51. Merge-order agreement remains pending. No dependency is invented. #6432 changes strength aggregation; #6769/#6778 change source-alias stub detection; none duplicates the new provider fallback. Tree merges are not combined runtime validation.Initial homebrew evidence
Used the unchanged main and module registries generated by the current-main relinker from the real Mari0 bundle, with
--windows --registry --trap-sdk-syscall-stubs --module-dir .and an added emptysce_moduledirectory. This is an indirect diagnostic workflow; it does not depend on the explicit-directory fix.Baseline: exit 2 at the first
strcmprecord, without an audit report. Fixed: exit 1 with a complete report of 561 references and 376 unique imports. All 376 remain absent against the supplied official v0.1.2 libraries, and ten.sprxdependencies remain missing; this fix enables diagnosis, not successful loading. The 126 official patched.prxinputs were read from the v0.1.2 release, not newly built. No library was executed.Package: Mari0 1.6.2 from websrv v0.34. Original layout and input hashes were checked before and after conversion; all inputs remained unchanged.
e0331ed5a22e7fe72843d268cd2c8b297d761a75b7c83d828f5cc58a82a72ce354bcc91670fd759bbd3488284dce351b6421cb83016e9ad8da6b5186da067a39128d2ab20b980b96decb1f20a9ed57d10cc922605259666b63efa52963920f0aNo generated homebrew executable was run. No game startup, gameplay, GPU, commercial-title, or physical-PS5 result is claimed.
Baseline relinker SHA-256:
a249e5a38abe9f338f09d6ea09182b5a39cd156dada6771984b85e315fda67c5. Tested registry-producing relinker SHA-256:a249e5a38abe9f338f09d6ea09182b5a39cd156dada6771984b85e315fda67c5.Official v0.1.2 PRX archive SHA-256:
b26e507e7cc01922312f4ca9c7b36ab333ed3a1d7048d676f20214cadf2d4895.Local logs, exact argv, per-file hashes, source patches, before/after failures and JSON results:
C:/Users/arjen/AppData/Local/Temp/anyps5-three-homebrew-20261014(homebrew-report.json, regression logs and CTest logs). The commands and decisive results are included above so this report does not depend on access to that local folder.Initial open PR comparison
Complete paginated inventory: 543 open PRs; main
d92a8dad390af1807cc91622b924252141816a5a. Complete changed-file lists and tree-only merge previews used that main as the explicit common base. This change merges with main and both other proposed changes. 18 relevant comparisons merge cleanly; 23 remain inconclusive because the other PR already conflicts with main. No conflict-free claim is made for those PRs.Exact compared heads and shared-file results
1800d3892bec857b254ca8c55e912e325e1c6711docs/user/USAGE.md09afe38730d7b9d40d57b8aa9e13777012d170a3docs/user/USAGE.md657f16ea286035a8615e2d56614a354cd4559e31docs/user/USAGE.md,tools/import_audit.py,tools/tests/test_import_audit.pyfcd477faf8fd0b4959f927db1f36590b317fa2b7tools/import_audit.py,tools/tests/test_import_audit.pyc690012ce1b6f73ab985c061509e7cbfb41127c2docs/user/USAGE.md62022e9362a27432e007cea132ecd9ca0909bb2bdocs/user/USAGE.md1f98703dd07a6257c84ea485594d92f5a8e84fbadocs/user/USAGE.md3339ddd3af77cfc336148966657c0d14e87b6ae1docs/user/USAGE.md569ff84ea2589a9c886dca37b5c9ddcea70049bddocs/user/USAGE.mde4036135ed0823b866969e6e7e688712b0f57fccdocs/user/USAGE.md9acd5dcc07d8149020bf50fbd5e09f7633a2613fdocs/user/USAGE.mdfd3c6796042ff61c599fa100a4e979a7be200658tools/import_audit.py,tools/tests/test_import_audit.pyacc228ab4a54fbdc0c816c22b52b82968e42649bdocs/user/USAGE.md2d14128a29ec22d25e0cb308777dd46894a73170docs/user/USAGE.md7ff22e76dc54f096f724381aef1a0c936b86cf9btools/import_audit.py,tools/tests/test_import_audit.pyd4fa8367daa0fd826853f440d23c3a604af43449docs/user/USAGE.md01f2073d06385ca2adb537d4996ad61e8255bd2edocs/user/USAGE.mdab0e383aca0fd9c8d873293ec0ffa170d750baa4docs/user/USAGE.md82de94cbedc5461de6af731b2a5a9838aca66e48docs/user/USAGE.mdb05e5cf8ecf4f410750c38a6f281f273babb816ddocs/user/USAGE.md2d6b8ef4341e21d78686bfba9da9a5c6bb14d629docs/user/USAGE.md1263d7065489fd6b7a15ebbecb1fcf0ee72d21f9docs/user/USAGE.md07713af3078fcb4e24a15c96bf15c3df4e2b071adocs/user/USAGE.mdbeb1ba79f2b3a2fc3882bf9d65553e4e754fc03ddocs/user/USAGE.mde4b37aa2e22c15e0e07341a7d8c18100cbae55c2docs/user/USAGE.md8c49e6d6960021ef72e1382d78b9b83914cc7f31docs/user/USAGE.md16b602cbfb29dcf3fc89b4ed67f228d65a27b20bdocs/user/USAGE.md66c8f11a47d64c2db6f514e409e94e91f5ac5309docs/user/USAGE.md92b12559c64489b556179b651ab4d966c29314ecdocs/user/USAGE.md67f9e72d1c663dbc8811cdb6c85753844be02381docs/user/USAGE.mdb339b4f6ca25b1b274555394e73a26f8c8d98347docs/user/USAGE.mdb447062d71dc1748d60edfdc0ec437aa597a5b7ddocs/user/USAGE.mdbad53434ecca2bf92ee67c43952598c6a108941edocs/user/USAGE.mdf9b136024e27d1463a70edee4952553628e5f289docs/user/USAGE.md5e296b79d94a6312c385890352d7f76e9a0d4380docs/user/USAGE.mdeb34b754de4b65b11c1f7d9beafe58a0d057c45ddocs/user/USAGE.mda6cb0cb5f277da258a661bb11792e04941f4c790docs/user/USAGE.md48bc3322b68ee855d9188f4eefd701e8007dd21bdocs/user/USAGE.md2636a486c2f71d880b69cadb706eac719f9cc670docs/user/USAGE.md67f94df827623df9210738a5ba3275b9a59f1c8bdocs/user/USAGE.md7b75adc8685edbffaa771e0d436543eaae694b8fdocs/user/USAGE.mdSource comparisons: #6171 handles module aliases, #6113 normalizes exported names, #5713 preserves Git stamp data, and #5571 handles stripped section tables. None accepts literal registry import names.
Checklist
main; no other open PR implements the same fix (overlapping changes listed above)